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[57] ABSTRACT 

The system enables a xiser to dispose information such as an 
icon to identify the article that the tiser posts from the client 
computer, on the display screen thereof, for example, at the 
position closer to an icon of an article that another user 
posted if the former user intends to express the opinion 
positively to the latter user's posted article. The server 
computer where many client computers can post articles and 
read the articles posted from other client computers, man- 
ages the coordinate data of the icons respectively disposed 
at the cUent computers according to the respective intentions 
of the articles and reproduces the icons on the display screen 
of a client computer, at the respective disposed positions so 
that a user at the client computer can instantly understand the 
intentions of the posted articles without reading the contents 
thereof. 

23 Claims, 11 Drawing Sheets 
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MULTIMEDIA ARCHITECTURE FOR tising is dependent upon prior interactions with the receiver 

INTERACTIVE ADVERTISING IN WHICH of the advertising. 

SECONDARY PROGRAMMING IS VARIED It is a further object of the present invention to provide a 

BASED UPON VIEWER DEMOGRAPHICS multi-media architecture wherein information obtained &om 

AND CONTENT OF PRIMARY 5 a viewer alters the flow of material to the viewer. 

PROGRAMMING It is yet another object of the present invention to provide 

on-demand primary programming with interactive advertis- 

This application is a continuation-in-part of a provisional 

application, U.S. Ser. No. 60/004,575. filed Sep. 29, 1995, ,! • t -*u u- * r*u * • . -j 
ai^P ivu, yv/,, H> '^^Itisa further object of the present mvention to provide a 

now pen mg. primary video stream which is interleaved with a secondary 

BACKGROUND OF THE INVENTION video stream at a lime in the primary video stream wherein 

the images displayed by the primary video stre agj are related 

1 . Field of the Invention t o.the contenU Lthe-secondary^videQ.striiSZ 
TTie present invention is directed to a multimedia archi- ,^ ^^^^^^^ ^^.^^^ ^^^^^ .^^^^^^^ 

tecture for providing interactive video programming to an ^ ^.^^^ ^^^^^^ commercial message breaks 

individual subscnber, for example to a television set, and, ^^^^ dependent upon the content of the primary video 

more particularly, directed to a multmiedia architecture that ^^^^^^ demographics of an individual subscriber, 

provides on demand programming mterleaved with second- , - . . 

ao' programming, such as advertising, to a subscriber iVlditional objerts and advantages of Uie mvention wdl be 

wherein the content of the secondary programming is depen- ^° set forth m part m the description which follows, and, m part, 

dent upon information gathered about the viewer. description, or may be learned by 

f.. ni.jA_* practice or the invention. 

2. Description of the Related Art *^ 

Current interactive applications are typically limited to ^hese and other objects of the invention are achieved in 

the providing ofon-demand movies, and games, via existing „ interactive multmiedia system comprising video server 

cable systems and telephone systems. For example, BeU " for providmg a prmiary video program mterleaved 

Atlantic is currently testing an on-demand movie system ^j"' secondary programming, the video server means being 

wherein subscribers arc able to. for a fee, select a movie. ''f P'*** '° mformation from a viewer during display 

from a list of movies. TTie selected movie is then transmitted ^e pnmary video programming and the «scondary 

, L u » 1 • . program, and database means for stonng the mformation 

to the subscnbers television set. m t - j i ■ i • • 

^ ^ . , . , . . about a viewer and analyzing the mformation to generate 

CurrenUy, commercial messages, i.e. advcrtismg are ^-^^^^ j^,^ ^y^^ ^^^^ ^^^^ ^y^^ ^^^^ ^^^^ 

delivered to a subscnber s TV from centralized broadcast „f - #u« 

„ .,, . . , ^ . , , content oi the secondary programming displayed to the 

facilities at either a national or local a level. The national and u^.^^ ^« tu^ ,»^„,«/^«7« 

, . , . , viewer based on the viewer data, 

local providers simply fiU-in designated, time-based, gaps in ^, . _ , . . , , . , . 

a primary video program with commercials. Tlie commer- 35 ^^^^f^^ }^^. Present mvention are also achieved in a 

cials are, at best, picked based upon a demographic profile ^^^^"^ of gathermg data from a viewer compnsmg display- 

of a probable cross-section of viewers of the primary video "^g, ^ P^'^^X program, mterleaving a secondary 

programming. U.S. Pat. No. 5,515,098 to Carles, incoipo- ^}^}^^^^^^ programmmg with the display of the prmiary 

rated herein by reference, sets forth a device and method for y^^^ pm^^m presenting questions to the viewer dunng at 

distributing commercial messages to an individually addres- 40 l^ast one of the pnmary video program and the secondary 

sable subscriber terminal on a network. In general, a data- rtiultimedia programming, receiving answers to the ques- 

base is generated using standard known poling techniques ^^^"^ f^^™ ^h^. ^^7^"? ^^e content of the 

having data regarding each subscriber to the network. This secondary multmiedia programmmg based on the answers, 

database is generated by, for example, mail-in responses, Objects of the present invention are further achieved in a 

telephone poling or in-person surveying. After the database 45 method of displaying comprising displaying a primary video 

has been generated, subscribers receive individualized com- program having a varying content, generating a video stream 

mercial messages during the normal gaps provided in the breaks file correlating the content of the primary video 

primary video programming. Carles further sets forth a program with the timing at which the content is displayed, 

method for matching the commercial messages to the appro- interleaving secondary multimedia programming with the 

priate viewer based upon the information stored in the 50 display of the primary video program, varying the content of 

database. the secondary multimedia programming based on the video 

However, to date, there has been no effective apparatus or stream breaks file, 
method for providing interactive advertising or other mul- Objects of the present invention are also achieved in an 
timedia services in conjunction with a primary video pro- interactive multimedia system comprising subscriber data- 
gram. Further, there have been no provisions for utilizing the 55 base means for storing subscriber data regarding demo- 
subscriber's television set to obtain statistical data regarding graphic along with preferences of a subscn*ber, a video 
the subscribers. Additionally, there has been no effective breaks database means which stores video breaks data 
apparatus or method for providing different commercials correlating a content of a primary video program with a 
based upon the images being displayed in the primary video timing at which the content exists within the primary video 
programming. 60 program, and video server means for providing the primary 

video program to an individual viewer interleaved with 

SUMMARY OF THE INVENTION secondary programming selected based on a correlation 

It is an object of the present invention to provide inter- between the subscriber data and the video breaks data, 
active advertising in conjunction with primary proeram- 

miug ^ ^ 65 ^^^^^ DESCRIPTION OF THE DRAWINGS 

It is another object of the present invention to provide an lliese and other objects and advantages of the invention 

interactive advertising system wherein the content of adver- will become apparent and more readily appreciated from the 
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following description of the preferred embodiments, taken movie, or the same movie at no charge with secondary 

in conjunction with the accompanying drawings of which: programming including interactive advertising an^^ 

HG. 1 is a block diagram of an interactive advertising qussliQns, requiring a response from the viewer, presented 

system in accordance with a first preferred embodiment of during the movie. The nature of the secondary programming 

the present invention. ^ presented to the viewer can vary based on a number of 

nz-i- U11-J* -J * A - determining factors, including the movie currently selected, 

FIG. 2 IS a block diagram of a video server m accordance , . r . • , j . > L I 

•.L i: . r J u J- * *' *' a past history of the movics Selected, the scenc in the currcnt 

with the first preferred embodiment or the present mvention. i—-. J., . , . -ui jt- ^ j • i 

^ ^ movie at which the advertising will be displayed, previously 

HG. 3 is a broad band processing unit for use in the video collected demographic information about the viewer In the 

server in accordance with the first preferred of the present interactive advertising, the advertising can change 

invention. based upon the viewer's response to the questions presented 

FIG. 4 is a block diagram of an interactive advertising during the interactive advertising in addition to any of the 

system in accordance with a second preferred embodiment previous mentioned determining factors, 

of the present invention. j^^^^ example, if a viewer selects an action movie as 

FIG. 5 is a block diagram of a set4op unit in accordance 15 the primary program, if at some point a truck is highlighted 

with the preferred embodiments of the present invention. jn a scene , secondary programnain g material, can l?e iat&iz. 

FIG. 6 is a flow chart of the top level process for the video leavedjw hich_ presents an^ dve^gi mcnt related to a'locaL 

server in accordance with either the first or second preferred truck dealer ^c arying a similar ino'ge r^f.t ruckJ lieing.shQ.^ 

embodiments of the present invention. in tne prun^^^^ ^ammg^^tematively, in an educational 

HG. 7 is a flow chart of a receiving process in the video settingTth'e^secondary programming can be tailored to a 

server in accordance with the preferred embodiments of the viewer's level of knowledge. For example, if a viewer has a 

present invention technical skill level (as determined by prior interactive 

FIG. 8 is a flow chart of the send video data process used >■ Primary programming can be interleaved with 

in the video server in accordance with the preferred embodi- „ ^'=°°'''^y programmmg matter explammg techmcal con- 

f . . 25 cepts presented m the primary program. Alternatively, if the 

ments of the present invention. , . ■ l i i u i • i 

^ viewer has a high level of technological awareness, the 

FIG. 9 is a diagram of an MHEG stream as used m the ^^^^y program would be interleaved only with minimal 

preferred embodiment of the present invention. secondary programming material. 

HG. 10 is a block diagram of an MPEG stream selector Alternatively, the secondary programming can be an 

and multiplexer as used within the video processor in 30 jnte ^ye presentation requesting input from the viewer in 

accordance with the preferred embodiments of the present thT ab^^ For example,- the viewer couiTbT 

invention. prompted to indicate whether, in fact, the viewer desires to 

FIG. 11 is a flow chart of an interleaving process in see the secondary programming. Alternatively, the viewer 

accordance with the preferred embodiments of the present can be requested to answer questions about the material seen 

invention. -^^ in the primary program to determine the amount of second- 
ary programming needed. This feature would be especially 

DESCRIPTION OF THE PREFERRED useful in the educational environment. Any information 

EMBODIMENTS collected during the secondary programming sessions can be 

Reference will now be made in detail to the present scored in a database and used in future sessions to further 

preferred embodiments of the present invention, examples ^fine the content of the secondary programming. The pre- 

of which are illustrated in the accompanying drawings. furred embodiments are descnbed as an interactive adver- 

wherein like reference numerals refer to like elements rising system, however the uses for such as system are not 

throughout. limited to advertising, but may be used, for example, for 

J ^ . J • • 1 ■ * education and entertainment. Further, while the primary and 

Interactive advertising involves interruptmg a primary 45 , . j L • -j -.1. 

.J . f , • 1 *u • secondary programs are descnbed as bemg videos, they may 

video programming stream, (or simply, the primary , ^ \ ^ , . t.* . . j- .1. 

V. ■ u * • / L u \ It also comprise animations, gra phics.,text >.aua iO-or any othe r 

program) bemg shown to a viewer (or subscriber) generally multimedia meoimn. — " 

comprising movies or educational material, at a predeter- — — =^ 

mined time, with one or more secondary video programming The primary program and secondary program are prefer- 

streams (or simply, the secondary programs), typically com- 50 ^^^^ encoded in the MPEGl format. The MPEGl format 

prising advertising, educational or other a>mmercial mate- requires a throughput averaging 1.5 megabits per second to 

rial. Additional facilities are provided to overiay textual Provide a fair quality picture. An El or Tl wirmg protocol 

and/or graphical material on the display and to receive input ^ well adapted to handle such throughput. If higher quality 

from the viewer. Thus, for example, the secondary video video is desired, an MPEG2 encoding can be used However, 

programming stream may be used to interactively request 55 MPEG2 encoding requires a higher average throughput, 

information from the viewer, such as demographics and typically in excess of 4 megabits per second, and as such 

shopping preferences. The information obtained from the would require an advance network such as ATM or other 

viewer may be utilized to preselect additional secondary broadband facilities. 

video programming streams for display at a future time. The FIG. 1 is a block diagram of an interactive advertising 

timing of the display of the secondary program may depend go system in accordance with a first preferred embodiment of 

upon the responses from the viewer and/or the content of the the present invention. In accordance with the first preferred 

primary program. embodiment, the primary and secondary programming 

For example, w hen a viewer selects a movie to view as th e material is delivered to a subscriber's house 500 via the 

primary program , he can be prompted to select whether he Internet 530, 

wishes to view the movie, for full charge without secondary 65 Each subscriber's house 500 is provided with a standard 

programs, i.e. advertising, the same movie for half charge known TV 502 which is attached to a set-lop unit 504, 

with some secon3ary^rogramming presented during the described hereinafter. The set-top unit 504 is connected to a 
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Standard cable modem 506. An off-the-shelf ?.C. 508 can be larly adapted to providing programming over existing cable- 
optionally provided The set-top unit 504 and the optional TV networks utilizing standard co-axial cabling to each 
EC. 508 provide the interface t o the inter active advertising subscriber's home, 

systcny ^Acable modem 506 is typically connected to a head in accordance with the preferred embodiments of the 
end 510 via an existing subscriber based cable network. The 5 present invention, the video server 520 may be realized on 
head end 510 comprises a cable modem 512 connected to the a modular multimedia architecture which utilizes multiple 
cable modem 506 and the subscriber's house 500. The cable application processing units (hereinafter APU). U.S. Pat. 
modem 512 is connected to a cable router 514 for dehvering Nq. 5,029,199 and U.S. Pat. No. 5,193,110, incorporated 
individualized programming material to each subscriber's herein by references, describe a distributed control and 
house 500. lo storage systems which may be modified, in accordance with 
The head end 510 is connected to the Internet 530. Also the present invention to produce a system for providing 
connected to the Internet are a series of video servers interactive advertising. In accordance with the first preferred 
520fl-520c. Each video server 520, discussed hereinafter, embodiment of the present invention APUs 10 and 12 are 
contains a content and subscriber database 522 and a pro- utilized to handle voice and application data. Additionally, 
gramming database 524. The programming database 524 three broad band processing units, 14, 16 and 18 (hereinafter 
stores the primary and secondary programming. The content BPU,) are utilized to handle video applications and video 
and subscriber database 522 stores statistical information data, which are outputted to a router 50. The router 50 
regarding each of the subscribers, including demographics, interfaces with the Internet 530 (or directly to a broad band 
network address and other related information. Additionally, network). As shown in FIG. 1, APU 10 and APU 12 share 
the content and subscriber database 22 stores content infor- 20 a SCSI disc 20. The APU 12 has access to a SCSI 22. 
mation regarding the primary and secondary programming Likewise BPU 14, BPU 16 and BPU 18 each have access to 
stored in the programming database 524. For example, with SCSI discs 24, 26 and 28, respectively which operate as the 
respect to video movies, the content and subscriber database programming database 524 and holds video programming 
522 would store timing and scene information regarding including movies and advertisement, 
each movie so as to enable a correlation between a content 25 APUs 10 and 12 are connected to a narrowband 
of a scene and the subscriber's demographic data. digital switching matrix 30 via Tl lines. The narrowband 
A standard Web server 540 is also connected to the digital switching matrix 30 is connected to a narrowband 
Internet 530. The Web server 540 serves out Web pages in private switched telephone network (hereinafter PSTN) 46 
HTML format to the Internet 530 in what is called the World via El lines 44. A master control unit controls the narrow- 
Wide Web. The Web server 540 is connected to a subscriber band digital switching matrix 30 and the router 50. The 
database 542 which is typically a table relating a subscrib- master control unit 32 is connected to the APU 10, the APU 
er 's IP address with his cable router addres s. . 12, the BPU 14, the BPU 16, the BPU 18 and the router 50 
In use, a subscriber will access thi Web server 540 using via an ethernet backbone 48. Preferably the master control 
-a stan jdard y{^]^ Browser via ^rrop^ena t-poagS^^piucr 32 has an associated SCSI disc array 36. Adatabase unit 
S OS^ombe s^-top unit 504, which iS" ttuTconfigurati on 38, for coUecting and analyzing data firom each of the 
wo uld' preterably comprise~an Internet ready device .J Using various components, is also connected to the ethernet back- 
t he Web Brow ser, the subscnber selects a primary program, bone 48. Preferably the database unit 38 is provided with a 
e.g. a movie'fro m the Web pages d isplayed b v the Web SCSI disk array 42. ^fhe database unit 38 acts as the content 
server46; which" he wishes to view mTmrWSOi. The Web and subscriber database 522 and stores various information 
server 540 accesses the subscriber database 542 to resolve about subscribers to the system including demographics, 
the subscriber's address. Thereafter, the Web server 540 past viewing history, past answers to interactive questions, 
directs the appropriate video server 520 to begin delivery of and other information as necessary. Additionally, the data- 
the primary program. The appropriate video server 520 can hase unit 38 can be programmed to make decisions regard- 
be selected based upon geographic location, the location of m when to interrupt a movie with advertising and what 
the subscriber's information, or even the location of the advertising programming to use. However, the BPUs can 
programming material. Thereafter, the video server 520 also be programmed for this task. 

delivers the primary and secondary program material to the An multiple integration unit 52 (hereinafter MIU) is 

cable router 514 and the head end 510 via the Internet 510. connected to the narrowband PSTN 46 using an SMDI 

//The cable router 514 routes the program material to the (simplified message desk interface) protocol. The MIU 52 is 

appropriate subscriber's house 500. A s the set -top unit 504 also connected to the ethernet backbone 48. The MIU 52 

provides a t wo w ay communication route willTtiie^video interfaces with other telephone systems, such as PBXS. A 

server 520, an interactiv^^iEwmg be carried out, call control adjunct (CCA) 54 is connected to narrowband/ 

as discus^Jhereinafter. " " -^.^-^r^ digital switching matrix 30 and to the narrowband PSTN 46\ 

WhUe the link to the set-top unit 504 from the headend 55 via the common channel system signaling 7 protocol 
510 has been described as a standard ca ble base subscriber (hereinafter CSS7). The CCA 54 is also connected to the 
system, one of ordinary skilFTh the ar't'wiUTccognizrth"at an ethernet backbone 48. A network interface unit 56 
asymmetric digital subscriber link may also be utilized. (hereinafter NIU) is connected to the ethernet 48 and inter- 
Further, various additio'nal'linKs^irasT Ti*oFan El data faces to other systems via FDDI (fiber distributed data 
line may also be utilized. 60 i^^^^^ace). 

FIG. 2 is a block diagram of a video server 520 in As shown in FIG. 2, the master control unit 32. the 

accordance with the first preferred embodiment of the database unit 38, the narrowband switching matrix 30, the 

present invention. Specifically, FIG. 2 sets forth a video MIU 52, the CCA 54, the NIU 56 and the router 50 are each 

server 520 which can form an interactive advertising system preferably provided with a backup redundancy unit, 
for use in outputting signals to a broad band public network, 65 The MCU 32 acts to assign an incoming call (in the case 

for example, ATM, 0C12, etc or a broadcast cable system as of voice mail, which can also include requests for video 

used in the preferred embodiment. This system is parti cu- programming), via the narrow band PSTN 46. The MCU32 
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directs requests for video programming, via the router 50 or services via the TV 502a. The set-top unit 504fl is connected 

the narrow band PSTN 46, to an appropriate APU or BPU to the TV 502fl via an A/B switch 118, which also interfaces 

over the ethernet backbone 48. Control data messages with the video conferencing unit 116. An alternative line 

between the MCU 32 and the APUs or BPUs, or between the with an amp 120 is provided for signal clarity. 
APUs and the BPUs, is routed over the ethernet backbone 5 FIG. 5 is a block diagram of a set-top unit 504 for use in 

48. an interactive advertising system in accordance with the 

rj„,- „ ,v preferred embodiments of the present invention. Preferably, 

Uurmc operation, a request tor voice or video services IS K . - * * i 

J • L J ncT-KT A£ *u 1 * * • »u the set-top unit 504 is configured around general purpose 

issued via the narrowband PSTN 46 or the Internet via the ^ ^. » j j * ^nn • * _c *u 

. • I. • computer, i.e. a standard PC. A CPU 122 interfaces with the 

router 50. If the services are voice services, such as voice ^^^^^^^^ .^^^^^^^ communication with the APUs, Air 

mail, the master control unit 32 routes the request to an 30 ^^^^^ ^.^^^^ ^^^^^^^ ^.^^ ^^^^ 

appropriate APU for service of the request by an APU via the ^-^ ^^^^^^ • ^^^^ ^ akevboa rd. mouse, 

narrowband digital switching matrix 30. If the request is for ^jLaninfrared remote control. AnTT^rimerf^e caTdl28 

video services, the master control 32 routes the request via p';^;id;r a signal to th TTV 502. Additionally, an overlay 

the ethernet backbone 48, to the appropriate BPU which processing unit 130 enables the CPU 122 to overlay text and 
provides services via the router 50, but may, with the is j^.^ ^„ ^ ^-^^^ ^^^'^^ ^ ^ ^24 is 

addition of an appropriate hnk, be output via the m narrow connected to the CPU 122 

band digital switching matrix 30. ™, , • •« iia u j . i 

^ ^ The overlay processing unit 130 can be used to display 

In general, the video programming provided by the BPUs j^^ts and graphics to a viewer in conjunction with, or 

may be either normal entertainment/educational program- independently of the primary or secondary programming 

ming or advertisement programming. The BPUs may access currently being displayed. For example, ajist of movies 

the database unit 38 to determine what type of programmmg c ould be displayed overtop a primary pro' gamshbwih g'k 

to provide to a subscriber. The programming may be divided continuou^-'StlWi r-'grTrm each 'Sovie! " 

among the BPUs or arranged dynamically. Further, a single Thereafter, w hen ^^^^n^^^^Hci^in'^^ 

massive disk array may be used and the requests for services with an inferred remote control umt, a keyboard or a'SS^Si, 

allocated across the BPUs based on the number of requests. the set-top unit 504 requests, f rom an assignedjsddeo,seiyeL 

FIG. 3 is a block diagram of a BPU 14 for use in the video SgQ, a secondary program showing a longer trailer regarding 

server 520 in accordance with the first preferred embodi- the selected movie while the overlay processing unit dis- 

ment of the present invention. The BPU 14 is provided vwth plays a message requesting the viewer to input whether he 

a CPU 64 which interfaces with the ethernet backbone 48 via wishes to receive the movie for free, with a lot of 

10 mbps ethernet adapters 66 and 68, The CPU 64 outputs advertising, for half the price with minimal advertising, or 

a signal via 100 mbps ethernet adapters 70 and 72 to the for full price with no advertising. Further, the overlay 

router 50. The BPU 14 is provided with an internal memory processing unit can be used during the advertising program- 

74. An array of SCSI adapters 76 interface with an associ- ming segments to take an order for the advertised product, 

ated array of SCSI discs 78 which preferably store the request input from the viewer regarding the viewers 

primary and secondary programming material. The CPU 64 responses to' the advertising, the desirability for future 

can, for example, comprise a 200 megahertz pentium pro- advertising regarding the product shown during the adver- 

cessor. In addition, each BPU can, for example, comprise a Using radio programming, or other demographic informa- 

standard PC mother board provided with the appropriate 10 tion. Additionally, the o verlay p r ocessin g unit_130 can be 

cards. adapted to disp la y other services , such as e-mail , video mai l. 

Although the first preferred embodiment of the present VQjLce^mail (wffhin an appropriate speech to text translator), 

invention has been described with respect to a specific faxes, etc. 

configuration, will be recognized at the first preferred In use, a viewer interacts with a set-top unit^ tp,,):cquest 

embodiment is not hmited to the specific configuration. For ^^j fe^^EI^g^y ^^'^ ' ' p'ru^^V^B^'sgcondf^lprogramming. 
example, while the present system has been described as 45 from the BPUs. The BPUs deliver video services through the 

outputting video signals to a router 50 for transmission over router 50 and the Internet 530 (in the first preferred 

the Internet 53, the router 50 can transmit information embodiment) or the digital switching service 96 (in the 

directly to the head end 510 via a broad band PSTN or the second preferred embodiment) to the appropriate set-top unit 

narrow band PSTN 46. However, the narrow band PSTN 46 504, which disj)laysjhe output on a 1 v ii4. I'HToveriay 
would have to provide an appropriate signal throughput for 50 processing unit 130 enables the display of menus and other 

handhng the video data. messages to the subscriber displaying information or 

FIG. 4 is a block diagram of an interactive advertising prompting for input, 
system in accordance with a second preferred embodiment FIG. 6 is a flow chart of the top level process for the video 

to the present invention. An APU 80 and an APU 82 server in accordance with either the first or second preferred 
interface via an ethernet network with bridge/CSU units 88 55 embodiment of the present invention. For the sake of clarity, 

and 90, which in turn interface with a digital switching the processor will be described with respect to the first 

system 96 (hereinafter DSS). Similarly, an APU 84 and an preferred embodiment. The process starts in step 1000. 

APU 86 interface with a DSS 96 via bridge/CSU units 92 Thereafter, the video server 520 waits until a connection 

and 94. The bridge/CSU units 88, 90, 92 and 94 preferably request is received in step 1002. Ilie connectionjceauest can 
output the via El fines. The output of the DSS 96 is 60 come via the Web server 540 over the Internet 530 or 

converted back into a ethernet signal via CSU/bridge units directly from the subscriber's P.C. 508. Once a connection 

98, 100, 102 and 104. Set-top units 5040-504^/ interface request is received, it is logged in step 1004, subsequently, 

with the CSU/bridge units 98, 100, 102 and 104, respec- a receiving process, see FIG. 7, is started in step 1006. After 

tively. The set-top units 504a-504d facilitate interaction the receiving process has completed the process returns to 
between the APUs and the subscriber via TVs 502a-502d. 65 step 1002. 

The set-top unit 504a is further connected to a video FIG. 7 is a flow chart of a receiving process in the video 

conferencing unit 116, which provides video conferencing server in accordance with the preferred embodiments of the 
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present invention. The receiving process is started in step FIG. 8 is a flow chart of the send video data process used 

2000. Thereafter, the video server 520 gets a message from in the video server in accordance with the preferred embodi- 

the set-top unit 504 in step 2002. In step 2004, the message ments of the present invention. The sending process starts in 

is reviewed to determine whether it contains a file name. If step 3000. In step 3002 data is read from the disk and sent 

the message contains a file name, the requested video file is 5 to the set-lop unit 504 via the Internet 530. Thereafter, in 

opened in step 2006. The process shown and described in step 3004, it is determined whether the set-lop unit has 

FIG. 7 is directed toward providing a primary video stream closed the connection. If the set-top unit 504 has not closed 

as the primary program and a secondary video stream as the connection, it is determined in step 3006 whether the end 

secondary program, but other types of streams may be used fii^ daU being sent has been reached. If the end of 

such as simple graphics or audio. The team stream indicates jq ^as not been reached, the process continues to read the 

that the data is sent and received as a linear, or non-linear, ^-^^^ ^^^^ ^^^^ gj^p 3Q02. Once the set-top has closed 

sequence of data. The term will become more clear in the connection in step 3004, or the end of file has been 

reference to FIG. 9 which shows a typical MPEG stream reached in step 3006, the process ends in step 3008. 

which preferably constitutes the primary and secondary r™. • , , - r j j • -.u 

. , *: 4n- c* J J J . • The mterleavmg of secondary video programming with 

video streams. Thereafter, a send video data process is . • l r j * 

started, see FIG. 8. in step 2008. In step 2009, a secondary " P^f^ 7'^"' programming can be performed at tmie 

. , . based mtervals, or based upon the content of the primary 

video program IS mteraclively mterleaved with the primary . ajj-.- n u r *i. 

. . ^ ^ . ,,,, /, . f ill r • video programming. Additionally, because of the presence 

video programming. When the end of file of the primary - ^ • r * . 

• . . . .1.1. .I l l- of a return communication path from the set-top unit 504, 

video programmmg is reached, the connected is closed in , . , . i i. *• * 

. lift 1 • • 11'* mil ihe secondary video programmmg can also be contmgent 

step 210 and the receiving process is ended in step 2032. j - a c .t, 

,r . ^^^A , , . r>. upon commands received from the set-top unit 504. 

If, m step 2004, the message does not contain a file name, Additionally, it is important to note, that the secondary video 

It IS deemed to be a command requestmg a change in the programming can also be considered primary video pro- 

flow of the primary video program, such as rewind, fast to further secondary video programming, thus in 

forward, pause, stop or the presentation of a secondary video ^q^^^ ^^^^^ ^ hierarchical tree of video programming to 

program. In step 2012, the data file of the requesting set-top 25 be presented to the subscriber. 

unit 504 IS opened, fhereafter m step 2014 it is determmed , , - , 

whether the message requires additional information, such general in order to allam mterleavmg based upon the 

that a question needs to be displayed on the TV 502. If a °^ '^e primary video prDgrammmg, the primary 

return question is to be displayed, an overlay is sent to the ' Pfog/imnwng must be analyzed prior to display, and 

overlay processing unit 130 in the set-top unit 504 in step 30 " f^™« °^ records generated setting forth the timmg and 

2016. niercaftcr, a response is received in step 2018 and the s*J'='=' m^ttnzl bemg di^layed durmg the 

process ends in step 2032. If the response can be handled by Pnn'^y prograinming. For example, various subject 

the set-top unit 504, such a process is initiated. However, if ""^"^^ *"= defined m relationship to the secondary video 

the response needs to be handled by the video server 520, the Programmmg and assigned numbers which are then stored 

response is treated as another message &om the set-top unit 35 ^ ""^ respective tmiings m the primary video 

and another receiving process is started with the response programmmg. 

being entered as the message. ¥IG. 9 is a diagram of an MPEG stream 600 as used in the 
If, in step 2014, the message does not require a return preferred embodiment of the present invention. An MPEG 
question, it is determined whether the viewer is viewing a stream 600 comprises a stream header 602 foUowed by a 
"must vieV' video in step 2020. A "must view" video may 40 ^"^^ ^^'^ packets 604fl to 604n. The stream header 602 
comprise, for example, advertising when the viewer has includes information on the time at which the first data 
indicated that he wishes to receive advertising in return for P^^^^^ ^ stream 600 is to be played. In the preferred 
a reduced cost of viewing the primary video programming. embodunents, the pnmary and secondary programs are 
Alternatively, a "must view'* video may also comprise, MPEG streams and are referred to as the pnmary video 
secondary video programming for which the advertiser has 45 secondary video stream, 
paid a premium fee to insure that a certain segment of the Each data packet 604 comprises a group of pictures 
population receives it. Alternatively, in an educational (GOP) 610 and a series of frames. In general, there are three 
setting, a "must view" video may comprise certain key typesof frames, ab-frame 612, an i-firame 614 andap-frame 
points of the educational material which the student is 616, The i-frame 614 comprises a full frame, while each of 
required to view. If the viewer is viewing a "must view" 50 the b-frame 612« and the p-frame 616^ contain predictive 
video, the process proceeds to step 2022 where it is deter- changes to the fuU frame, contained in the i-frame 614. 
mined whether the command requests a forward shift in the FIG. 10 is a block diagram of an MPEG stream selector 
current video stream or an entirely new video stream. If and multiplexer 700 as used within the video processor in 
command does request a forward shift or change from the accordance with the preferred embodiments of the present 
current video stream, the process is waited until the current 55 invention. Preferably, the MPEG stream selector and mul- 
"must view" video stream is completed in step 2024, tiplexer 700 is realized in software in each BPU and 
Once it has been determined that the viewer is not viewing executed by the CPU 64. The MPEG stream selector and 
a "must view" video, the process proceeds to step 2026 multiplexer 700 is provided with a parameter buffer 702 
wherein the requested video stream is opened to the (preferably in the memory 74) and a subscriber parameters/ 
requested position in the command. At this point, the 60 MPEG stream content breaks database 704 (preferably 
requested video data is sent in a sending process, see FIG. formed in the array of SCSI disks 78). The MPEG stream 
8. Thereafter, in step 2028, secondary video programming is selector and multiplexer 700 receives and multiplexes a 
interactively interleaved into the primary video program- MPEG content stream 710, i.e. the primary video stream, 
ming. In step 2020, when the end of file of the primary video and MPEG commercial segments 720a-720/i, i.e, the sec- 
programming is reached, the set- lop is updated and the 65 ondary video stream. A single MPEG output stream 730 is 
connected is closed. Subsequently, the process ends in step output by the MPEG stream selector and multiplexer 700. 
2032. An input line 740 is provided to enable the BPU or set-top 
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unit 504 to control the selection of the primary and second- 
ary video streams in response to commands from the sub- 
scriber. 

FIG. U is a flow chart of an interleaving process in 
accordance with the preferred embodiments of the present 
invention. The process starts in step 4000. TTie header of the 
primary video stream is copied into the parameter buffer in 
step 4002. Concurrently, the header of the primary video 
stream is delivered to the output channel 730 in step 4004. 

In step 4006, the next break point in the primary video 
stream is determined. The break point is the point at which 
the secondary video stream should be interleaved with the 
primary video stream. This decision is made based upon 
information contained in the subscriber parameters/MPEG 
stream content breaks database 704 or in response to a 
command from the subscriber. Typically, the subscriber 
parameter/MPEG stream content breaks database 704 will 
be generated prior to the showing of the primary video 
stream for each individual subscriber, by correlating the 
demographic data contained in the content and subscriber 
database 522 with timing and subject information which has 
been previously generated about the primary video program- 
ming. While the interleaving process is being shown and 
described as being a hnear process, it will be recognized by 
one of ordinary skill in the art that parallel processing can be 
used, wherein the break points are determined concurrently 
with the displaying of the primary video stream. 

In step 4008, the primary video stream is sent to the output'' 
channel. In step 4010, it is determined whether the end of file 
of the primary video stream has been reached. If the end of 
file has been reached, the process ends in step 4024. 

If the end of file has not been reached, it is determined 
whether a break point has been reached in step 4012. If a 
break point has not been reached, the process returns to step 
4008 and continues to send primary video stream to the 
output channel 730. 

However, if, in step 4012, a break point has been reached, 
the primary video stream is stopped in step 4014, 
Subsequently, in step 4016, a reset code is sent to the set-top 
unit. TTie re after, in step 4018, a secondary video stream, e.g. 
the MPEG commercial segment 720fl-720^> is sent to the 
output channel 730. 

Once the end of file of the secondary video stream has 



with respect to an interactive advertising system, one of 
ordinary skill in the art will also recognize that it can be 
utilized in a variety of educational and entertainment sce- 
narios. 

5 Although a few preferred embodiments of the present 
invention have been shown and described, it would be 
appreciated by those skilled in the art that changes may be 
made in these embodiments without departing from the 
principles and spirit of the invention, the scope of which is 

30 defined in the claims and their equivalents. 
What is claimed is: 

1. An interactive multimedia system comprising: 

a video server providing a primary video program inter- 
leaved with secondary programming, said video server 
receiving infonnation from a viewer during display of 
the primary video program and the secondary program- 
ming; and 

a database storing the information from the viewer and 
analyzing the information to generate viewer data 
which causes said video server to vary the content of 
the secondary programming displayed to the viewer 
based on the viewer data at a timing based upon a 
content of the primary video program. 

2. An interactive muhimedia system, as set forth in claim 

1, further comprising: 
an interface which receives the primary video program 

and the secondary programming from said video server 
and displays the primary video program and the sec- 
ondary video programming on a monitor to the viewer. 

3. An interactive multimedia system, as set forth in claim 

2, wherein said interface overlays the primary video pro- 
gram and secondary programming with graphics. 

4. An interactive multimedia system, as set forth in claim 

3, wherein the graphics are in the form of questions to which 
the user supplies answers. 

5. An interactive multimedia system, as set forth in claim 

4, wherein said interface receives the answers supplied by 
the user and transmits the answers to said database. 

6. An interactive multimedia system, as set forth in claim 

5, wherein said database uses the answers supplied by the 
user to vary the content of the secondary programming. 

7. An interactive multimedia system, as set forth in claim 
2, wherein said interface gathers information from the 
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been reached in step 4020, a new stream header 602 is 45 viewer and transmits the information to said video server. 



generated in step 4022. A copy of this header is stored in the 
parameter buffer 702. The new header contains new timing 
information regarding the playing of the remainder of the 
primary video stream. After the new header has been stored, 
in step 4022, the process returns to step 4000 and the next 50 
break point is determined. 

As shown and described above, the preferred embodi- 
ments of the present invention enable an individualized 
video programming stream to be transmitted to a subscrib- 
er's house. The individualized video programming stream 55 
comprises a primary program, preferably formed of a pri- 
mary video stream and a plurality of interleaved secondary 
programs, preferably formed of secondary video streams. 
The secondary video programs are interleaved, based upon 
input received from the viewer, subject matter being dis- 60 
played in the primary material, and/or demographic and 
personal data regarding the subscriber stored in a database. 
Additionally, as the system is interactive, the data regarding 
each viewer stored in the database can be updated in 
response to questions, by the overiay processing unit 130, 65 
presented during the presentation of either the primary or 
.secondary material. While the system has been described 



8. An interactive multimedia system, as set forth in claim 
1, wherein the primary video program is a video stream. 

9. An interactive multimedia system, as set forth in claim 
8, wherein the primary video program is a movie. 

10. An interactive multimedia system, as set forth in claim 
1, wherein the secondary programming comprises an adver- 
tisement. 

11. A method of gathering data from a viewer comprising: 
displaying a primary video program; 

interleaving secondary multimedia programming with the 
display of the primary video program; 

presenting questions to the viewer during at least one of 
the primary video program and the secondary multi- 
media programming; 

receiving answers to the questions from the viewer while 
the questions are being presented; and 

varying the content of the secondary multimedia program- 
ming based on the answers. 

12. A method of displaying comprising: 

displaying a primary video program having a varying 
content; 
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generating a video stream breaks file correlating the 
content of the primary video program with the timing 
at which the content is displayed; 

interleaving secondary multimedia programming with the 
display of the primary video program; 5 

varying the content of the secondary multimedia program- 
ming based on the video stream breaks file. 

13. An interactive multimedia system comprising: 

a subscriber database for storing subscriber data regarding 
demographics along with preferences of subscribers; iq 

a video breaks database which stores video breaks data 
correlating a content of a primary video program with 
a timing at which the content exists within the primary 
video program; and 

a video server providing the primary video program to an 15 
individual subscriber interleaved with secondary pro- 
gramming selected based on a correlation between the 
subscriber data and the video breaks data. 

14. An interactive multimedia system comprising: 

a database which stores information about a content of a 20 
primary video program based on time; 

a video server which outputs a signal representative of the 
primary video program interleaved with a secondary 
video program, the interleaving of the secondary video 
program being at a timing responsive to the informa- 25 
tion. 

15. An interactive multimedia system comprising: 
video server means for providing a primary video pro- 
gram interleaved with secondary programming, said 
video server means being adapted to receive informa- 30 
tion from a viewer in response to display of the primary 
video programming or the secondary programming; 
and 

database means for storing the information and analyzing 
the information from the viewer to generate viewer data 35 
which enables said video server means to vary the 
interleaving and the content of the secondary program- 
ming displayed to the viewer. 

16. An interactive multimedia system comprising: 

a database which generates viewer data based on infor- 40 
mation about a viewer; 

a video server providing a primary video program inter- 
leaved with secondary programming selected based on 
the viewer data and at a timing based on a content of the 
primary video program, said video server receiving 
information about the viewer from the viewer and 
supplying the information about the viewer to said 
database. 
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17. An interactive multimedia system, as set forth in claim 
16, wherein: 

said video server receives information about the viewer 
from the viewer in response to display of one of the 
primary video programming and the secondary pro- 
gramming. 

18. An interactive multimedia system, as set forth in claim 
16, further comprising: 

an interface receiving the primary video program and the 
secondary programming from said video server and 
displaying the video programming on a monitor to the 
viewer. 

19. An interactive multimedia system, as set forth in claim 
18, wherein said interface overlays the primary video pro- 
gram and secondary programming with graphics. 

20. An interactive multimedia system, as set forth in claim 
16, wherein the primary video program is a video stream. 

21. An interactive multimedia system, as set forth in claim 
20, wherein the primary video program is a movie. 

22. An interactive multimedia system, as set forth in claim 
16, wherein the secondary programming comprises adver- 
tisement. 

23. A video delivery system utilizing the Internet, com- 
prising: 

an interface unit, provided at a subscriber *s location, 
receiving and displaying subscriber video program- 
ming via the Internet and providing information via the 
Inlemet; and 
a video server connected to the Internet including: 
a programming database storing primary video pro- 
grams and secondary programming, said program- 
ming database including routines for providing sub- 
scriber video programming by interleaving a primary 
video program with secondary video programming; 
and 

a content and subscriber database storing the informa- 
tion from the viewer and analyzing the information 
to generate viewer data for the routines used to create 
the subscriber video programming, the viewer data 
indicating the content of secondary programming to 
displayed being varied based on the viewer data and 
a timing of the display of the secondary program- 
ming being varied based upon a content of the 
primary video program. 
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